Generally speaking, for the baselines, you need to compare with other (i.e., state-of-the-art) and compare with yourself (i.e., component analysis or ablation study).
Compare with others
How to select baselines?
- Select the baselines from top conferences. You can refer to related paper published on recent top conferences and find out which baselines they compare with. The intersection of their baselines should be the most popular ones.
- The selected baselines should be discussed in the related work in your paper.
- The selected baselines should cover at least several ones from the most recent top conferences.
- The selected baselines should cover the researchers who are very famous in this field or has many publications in this field. If you do not cite his/her paper and your paper unfortunately goes under his/her review, then you are doomed.
Could we directly copy the results from previous papers?
- Carefully compare your experiments settings with those described in previous papers. The experimental settings include training/testing split, input format (e.g., image size), evaluation metric, backbone network, etc. If they are exactly the same, just copy the results.
- Otherwise, you need to re-run their methods with exactly the same experimental settings as yours for fair comparison.
Need we implement the baselines?
- Search the code online and contact the authors for code.
- If you could not get the code, you need to implement the baseline by yourself according to the details provided in the paper. Theoretically, it is impossible to completely re-implement the baseline unless the method is frustratingly easy (e.g., 10 lines of matlab code), so just follow your understanding and implement a reasonable version.
Compare with yourself
Why is it necessary?
- Because you need to understand which component of your method really works.
- If you do not compare with yourself, you provide a perfect reason for reviewers to reject your paper.
How many special cases do we need?
- That mainly depends on the technical contribution of your paper. If you claim regularizer XXX or strategy XXX or subnetwork XXX is proposed by yourself and very effective, you have to verify that in the experiments.
- For some naive special cases, you may just need to set certain hyper-parameter as 0 or freeze some components in your network, so the experiments will be quite simple. For other advanced special cases, that will take some more work.
Checklist
Please select and compare with baselines meticulously. You can summarize and check your baseline information in the following table:
